Method for user recognition and emotion monitoring based on smart headset

ABSTRACT

A method for user recognition and emotion monitoring based on a smart headset is provided. —The smart headset includes an earplug part and a main body, wherein the earplug part is provided with a first microphone and a wearing detection sensor, and a housing of the main body is provided with a signal amplification circuit, a communication module, and a microcontroller. The wearing detection sensor is to detect whether a user wears the smart headset, and the first microphone is to obtain a sound signal in an ear canal. The sound signal is amplified, and then is outputted to the microcontroller. The amplified sound signal is transmitted by the microcontroller to a smart terminal paired with the smart headset to extract a heart sound signal characteristic, and legality of—identity of the user is validated and emotional state of the user is inferred according to the heart sound signal characteristic.

CROSS REFERENCE TO THE RELATED APPLICATIONS

This application is the national stage entry of InternationalApplication No. PCT/CN2021/079115, filed on Mar. 4, 2021, which is basedupon and claims priority to Chinese Patent Application No.202010535561.7, filed on Jun. 12, 2020, the entire contents of which areincorporated herein by reference.

TECHNICAL FIELD

The present disclosure relates to the field of identity authenticationand emotion monitoring technologies, and more particularly, to a methodfor user recognition and emotion monitoring based on a smart headset.

BACKGROUND

With the development of smart wearable devices, headsets are accepted bymore and more people. However, the headsets also have privacy functionssuch as communications and voice chats, to protect user personalinformation, headset-based identity authentication functions haveattracted widespread attention in the industry. At present, there aretwo main limitations on the headsets, namely, requirements for activeinteraction, and easiness of being stolen and pilfered. Therefore, userauthentication needs to be performed when a smart headset is in use.That is, it is needed to determine whether a user of this headset is aregistered user (owner), and after it is determined that the user ofthis headset is the owner of this headset, it is even further needed tounlock devices such as a mobile phone or a computer paired with thisheadset. Therefore, the user authentication will satisfy the user'sstrong demands for product privacy and protection of usage rights, andcan significantly improve intelligent autonomy of products.

In addition, as life pressure continues to increase, more and morepeople are emotionally unstable, depressed for a long time, and sufferfrom depression, anxiety, etc. Therefore, it is important to timely andcontinuously monitor emotions and provide emotional relief.

In the existing technologies, technical solutions for implementing theidentity authentication mainly include following methods.

1) Identity authentication based on information secrets. That is,identity authentication is performed based on information that the userknows, such as static passwords, etc. This method is simple in use anddeployment, but is lower in security, and authentication information iseasy to be stolen.

2) Identity authentication based on trusted objects. That is, identityauthentication is performed based on information or device owned by theuser, such as smart cards, SMS passwords, dynamic passwords, USB KEY,OCL, and Bluetooth locks, etc. Network security of this method is higherthan the identity authentication based on information secrets. Thismethod lacks convenience and requires the user to carry the trustedobjects. When the trusted objects are accidentally lost, guarantee ofsecurity of this method is lost.

3) Identity authentication based on biological characteristics. That is,identity authentication is performed based on biological characteristicssuch as measurable and unique bodies or behaviors, for example, commonretina recognition, iris recognition, fingerprint recognition, palmrecognition, face recognition, voice recognition, signature recognition,vascular texture recognition, human odor recognition, DNA recognition,electrocardiograph (ECG) recognition, electroencephalogram (EEG)recognition, etc. This method is convenient, unique andnon-reproducible, and has the highest security. However, this methodneeds large authentication devices, which are inconvenient to carry.Furthermore, a range of application of the authentication devices islimited, and this method cannot be implemented on the smart headsets.

4) Identity authentication for wearable devices based on physiologicalcharacteristics and behavioral characteristics. For example, there is amethod for identity authentication based on EEG biologicalcharacteristics in ears. In this method, electrodes are added into anin-ear part of the headset to collect the EEG, and identity recognitionis implemented through a support vector machine categorizer. For themethod of collecting the EEG, the main problem is that collectingdevices are complicated and collecting conditions are demanding. Forexample, it is not only required to ensure that the in-ear part of theheadset is in perfect contact with the ear canal, but also required toattach an extra electrode to an earlobe part as a reference electrode,which is inconvenient in actual use. For another example, there is alsoa method for identity authentication using echoes of playing specificaudio to obtain unique physical and geometric characteristics of the earcanal. In this method, a microphone is added nearby a speaker of theheadset to collect the echoes generated by the ear canal when the audiois played. These echoes contain the unique physical and geometriccharacteristics of the ear canal. After characteristic extraction iscompleted, the identity recognition is implemented through the supportvector machine categorizer. For the method of audio echo, the mainproblem is that a specific piece of audio needs to be played first. Thispiece of audio may be a swept-frequency signal, a fixed-frequency sinewave or a specific audio frequency. However, in either way, the userwill be disturbed by the sound being played.

In the existing technologies, technical solutions for implementingemotion monitoring mainly include following methods.

1) Emotion recognition based on facial expressions. This method requiresthe use of a camera to continuously track changes of facial expressions.This method is expensive, requires active cooperation from users, hasprivacy issues, and is easy to pretend such that it is unable to detecttrue internal emotions.

2) Emotion recognition based on speech signals. In this method, semanticcontents of speech or rhythms of speakers are analyzed. This method alsohas the risk of leaking the users' speech contents and is greatlyaffected by the differences in habits of individual expressing emotions.Furthermore, this method is also easy to pretend such that it is unableto detect true internal emotions, requires the users to speak to provideemotion monitoring, and requires the users to cooperate for use.

3) Emotion recognition based on physiological signals. For example,common physiological signals include electroencephalogram (EEG) signals,electromyography (EMG) signals, skin electric signals,electrocardiograph (ECG) signals, pulse signals, and respiration (RSP)signals, etc. This method is more related to people's internal emotionalstates. This is because human physiological signals are only dominatedby autonomic nervous systems and endocrine systems. However, in order tomeasure accurate physiological signals, devices used in this methodgenerally are bulky and inconvenient to carry, which hinders the users'daily activities.

4) Emotion recognition based on multiple modalities. This methodintegrates two or more different signals of the above technologies.Although this method has the advantage of accuracy, it also has thedisadvantages of the above methods.

In summary, the devices for identity authentication and emotionmonitoring in the existing technologies mainly have the followingdisadvantages. These devices are inconvenient to carry. It is easy toleak the users' privacies because of poorer information security. Thesedevices have poor robustness, and results of identity authentication andemotion monitoring are not accurate enough, and are susceptible toexternal factors.

SUMMARY

An objective of the present disclosure is to overcome the defects of theexisting technologies by providing a method for user recognition andemotion monitoring based on a smart headset. After a user wears theheadset, autonomous identity authentication and emotion monitoring canbe implemented.

Technical solutions of the present disclosure are as below. There isprovided a method for user recognition and emotion monitoring based on asmart headset. The smart headset includes an earplug part and a mainbody, the earplug part is provided with a first microphone and a wearingdetection sensor, and a housing of the main body is internally providedwith a signal amplification circuit, a communication module, and amicrocontroller. The wearing detection sensor is configured to detectwhether a user wears the headset properly, and the first microphone isconfigured to obtain a sound signal in an ear canal. The sound signal isamplified by the signal amplification circuit, and then the amplifiedsound signal is outputted to the microcontroller. The amplified soundsignal is transmitted by the microcontroller via the communicationmodule to a smart terminal paired with the smart headset to extract aheart sound signal characteristic, and legality of the user's identityis validated and the user's current emotional state is inferredaccording to the heart sound signal characteristic.

Compared with the existing technologies, advantages of the presentdisclosure are as below. In addition to providing functions of a generalheadset, the smart headset system provided by the present disclosure canalso autonomously perform identity authentication and emotion monitoringafter the user wears the headset, which not only can protect the user'sprivacy but also can continuously track the user's internal emotions inreal time. Furthermore, the smart headset is lower in hardware costs andconvenient for use, and can accurately and quickly perform identityauthentication and can continuously perform emotion monitoring duringapplication.

Other characteristics and advantages of the present disclosure willbecome apparent from the following detailed description of exemplaryembodiments of the present disclosure with reference to accompanyingdrawings.

BRIEF DESCRIPTION OF THE DRAWINGS

The accompanying drawings herein are incorporated in and constitute apart of this specification, illustrate embodiments of the presentdisclosure and, together with the specification, serve to explain theprinciples of the present disclosure.

FIG. 1 is a schematic diagram of an integral structure of a neck-hangingheadset according to one embodiment of the present disclosure;

FIG. 2 is a schematic structural diagram of an earplug part of a smartheadset according to one embodiment of the present disclosure;

FIG. 3 is a flowchart showing user registration according to oneembodiment of the present disclosure;

FIG. 4 is a flowchart showing identity authentication and emotionmonitoring according to one embodiment of the present disclosure; and

FIG. 5 is a flowchart showing identity authentication and emotionmonitoring algorithms according to one embodiment of the presentdisclosure.

DETAILED DESCRIPTION OF THE EMBODIMENTS

Various exemplary embodiments of the present disclosure will now bedescribed in detail with reference to the accompanying drawings. It isto be noted that the relative arrangement, numerical expressions, andnumerical values of the components and steps set forth in theseembodiments do not limit the scope of the present disclosure unlessotherwise specifically stated.

The following description of at least one exemplary embodiment isactually merely illustrative, and in no way serves as any limitation onthe present disclosure and application or use thereof.

Technologies, methods and equipment known to those of ordinary skill inthe related art may not be discussed in detail, but where appropriate,the technologies, methods and equipment should be considered as part ofthe specification.

In all examples shown and discussed herein, any specific values shouldbe interpreted as merely exemplary and not limiting. Therefore, otherexamples of the exemplary embodiment may have different values.

It is to be noted that similar reference numerals and letters indicatesimilar items in the following accompanying drawings. Therefore, once anitem is defined in one drawing, there is no need to discuss this itemfurther in subsequent drawings.

The smart headset provided by the embodiments of the present disclosuremay be of different types or forms. For example, the smart headset maybe an in-ear, ear-hanging or neck-hanging headset. The smart headset asa whole includes an earplug part and a headset body. The earplug partincludes a speaker and a microphone, etc. The headset body includes ahousing, a key or touch panel arranged on the housing, and a circuitboard encapsulated in an inner cavity of the housing, etc.

Reference is made to FIG. 1 and FIG. 2 by taking an in-ear headsetprovided with a neck-hanging coil as an example to give a description,wherein the earplug part 100 includes an headset speaker 1, a microphone2 hidden in the earplug, and a wearing detection sensor 3. In thisembodiment, the headset body 200 is a neck-hanging collar, and itshousing is provided with a play/pause/power key 4, a volume up key 5, avolume down key 6, a circuit board 7 encapsulated in the housing, apower indicator light 8, a Type-C interface 9, and a microphone 10.These control keys may be physical control keys or a touch panel thatcan be operated by touch.

In the embodiment as shown in FIG. 1, the smart headset includes atleast two microphones, where the microphone 2 is configured to obtain asound signal in an ear canal, and the microphone 10 is a microphone ofthe headset itself. The housing of the headset body 200 may be furtherprovided with a microphone key for activating the sound detectionmicrophone in the ear canal.

In the embodiment as shown in FIG. 1, an internal microphone and anexternal microphone are used, and the internal microphone is configuredto collect a heart sound signal. However, it is to be noted that thepresent disclosure is not limited to this dual-microphone solution. Inother embodiments, a technology of integrating the speaker with themicrophone may also be used. That is, there is only one electronic unitinside the earplug, and this electronic unit not only can be used as thespeaker, but also can be used as the microphone.

The circuit board 7 can integrate with a sound signal amplificationcircuit, a communication module, and a microcontroller (MCU). Thesemodules may exist in the form of independent modules or may beintegrated on the same circuit board. The present disclosure is notlimited thereto. The communication module is configured to communicatewith external smart terminals in a wired or wireless manner (such asBluetooth). The external smart terminals include, but are not limitedto, mobile phones, tablet computers, MP3 and other electronic devices.

The wearing detection sensor 3 is configured to detect whether theheadset is properly worn, and the wearing detection sensor 3 may be acontact detection sensor, a capacitive sensor, or a photoelectricsensor, etc. For example, a capacitive sensing sensor is arranged, andwhen the earplug part is close to an ear, it is determined whether thein-ear headset has been properly worn according to the capacitancevariation in the capacitive sensing sensor. For another example, thephotoelectric sensor is arranged to detect whether the in-ear headset issuccessfully worn according to the variation in an output signal causedby the variation in light intensity. Preferably, in the embodiments ofthe present disclosure, the photoelectric sensor is arranged obliquelybelow the in-ear earplug part. In short, the working principle andworking process of the smart headset provided by the embodiments of thepresent disclosure are as below. After a user inserts the in-ear headsetinto the ear canal, the photoelectric sensor on the headset detects thatthe user wears the headset, and a sound signal in the ear canal iscollected using the microphone arranged in the earplug part. The speakercan also play music normally when the headset collects sound in the earcanal. The sound signal collected by the microphone from the ear canalis amplified by an amplification circuit, and then the amplified soundsignal is outputted to the microcontroller. The sound data obtained istransmitted by the microcontroller via the communication module to anexternal smart terminal, such that the smart terminal processes (such asframing, filtering, etc.) the sound signal in the ear canal, andextracts a heart sound signal to obtain a heart sound signalcharacteristic, and inputs the heart sound signal characteristic into apre-trained identity recognition model to categorize the heart soundsignal characteristic and determine whether the individual heart soundsignal characteristic belongs to a registered user (i.e., a legitimateuser), and compares a determination result with a previously obtainedresult of a registered user to implement user authentication and deviceunlocking functions. After the device is unlocked, the heart soundsignal characteristic is inputted into a pre-trained emotion recognitionmodel to categorize the heart sound signal characteristic and obtain theuser's current emotional state, and the user's current emotional stateis recorded and archived to generate an emotion monitoring report.

Specifically, the process of performing identity authentication andemotion monitoring using the smart headset provided by the presentdisclosure includes a registration process of use by the user for thefirst time (to achieve identity authentication and emotion monitoring,the user is required to enter his/her heart sound data for modeltraining before using the smart headset) and a process of identityauthentication and emotion monitoring in actual use.

With reference to FIG. 3, the registration process of use by the userfor the first time includes following steps.

In Step S310, the headset is activated to start an APP and performpairing with the headset on a mobile phone.

For example, the user activates the headset, starts the APP of themobile phone and performs Bluetooth device pairing.

In Step S320, when prompted, the headset is worn, and it is waited forentering the heart sound signal characteristic and completing thetraining of the identity recognition model and the emotion recognitionmodel.

When prompted, the headset is worn, and it is waited for entering theheart sound characteristic and completing the training of therecognition models. It is to be understood that the identity recognitionmodel and the emotion recognition model may be trained online or trainedoffline, and then are integrated into the headset to speed up theregistration process.

It is to be noted that the user may select the emotion recognition modelusing a general-purpose emotion recognition model built in a system fortraining the emotion recognition model instead of using the user's ownheart sound signal, to speed up the registration process.

The general-purpose emotion recognition model may be obtained bytraining pre-collected heart sound characteristics and emotional statetags of other different users (not currently registered users).

In Step S330, the user registration is completed.

After the user registration is completed, the smart headset can be usedby the user normally.

With reference to FIG. 4, the process of identity authentication andemotion monitoring in actual use includes following steps.

In Step S410, after the sensor on the headset detects that the headsethas been worn properly, the headset starts to collect the heart soundsignal in the ear canal.

In the actual use, the user only needs to wear the headset properly, andthe headset will automatically recognize whether the headset is in aproper wearing state through the photoelectric sensor.

In Step S420, the amplification circuit amplifies the heart soundsignal, and the amplified heart sound signal is collected by the MCU andis transmitted to a smart mobile phone through the communication module.

The heart sound signal obtained by the microphone is amplified by theamplification circuit, and then the amplified heart sound signal iscollected by the MCU. Next, the amplified heart sound signal istransmitted to the smart mobile phone through the communication modulesuch as a Bluetooth module.

In Step S430, the smart mobile phone processes the received signal,extracts a heart sound signal characteristic, and transmits the heartsound signal characteristic to the trained identity recognition model tocategorize the heart sound signal characteristic.

After receiving the data, the smart mobile phone processes the data by,for example, framing, filtering and characteristic extraction, and thentransmits the processed data to the pre-trained identity recognitionmodel to categorize the processed data.

In Step S440, it is determined whether the user is a legitimate user.

The obtained result is matched with a stored legitimate user tag todetermine whether the user is the legitimate user.

If the user is the registered legitimate user, the headset is unlockedand the mobile phone is unlocked (Step S450). If the user is determinedto be an illegitimate user, an error prompt tone may be played (StepS460).

In Step S470, after the identity authentication is completed, the heartsound signal characteristic is transmitted to the trained emotionrecognition model to categorize the heart sound signal characteristic.

If the headset is unlocked and the mobile phone is unlocked (Step S450),the characteristic extracted in Step S430 is transmitted to thepre-trained emotion recognition model to categorize the characteristic.After obtaining the output, the smart mobile phone will archive theuser's current emotional state and generate an emotion monitoring report(Step S480).

In one embodiment, with reference to FIG. 5, the process of identityauthentication and emotion monitoring algorithms includes followingsteps.

In Step S510, framing is performed on an original heart sound signal.

For example, the mobile phone frames the received data using a Hammingwindow function.

In Step S520, the signal is filtered and a heart sound characteristic isextracted.

The sound signal in each window is filtered to obtain the heart soundsignal. The user is likely playing music at the same time, and thus thecollected original sound signal may likely be a signal obtained bymixing heart sound and other sounds. The sound signal needs to befiltered to extract a signal corresponding to the heart sound. Anadult's heart rate ranges between 40 BPM and 100 BPM, and may reach upto 220 BPM during exercise. A sampling rate of the microphone is 44.1KHz, and it has been verified that noise can be effectively filtered outby arranging a reasonable band-pass filter. In addition, the heart soundsignal may also be extracted by using in combination with waveletfiltering and mean filtering, etc.

In the embodiments of the present disclosure, the sound data is framedby using the Hamming window function, and then the sound signal in eachwindow is filtered. In this way, the signal corresponding to the heartsound can be accurately extracted, and the accuracy of subsequentcategorization can be enhanced.

Further, characteristic extraction is performed on the heart soundsignal. For example, a time domain characteristic and a frequency domaincharacteristic are extracted using time-frequency transformtechnologies. The time-frequency transform technologies include fastFourier transform, short-time Fourier transform, Wigner-VilleDistribution (WVD), and wavelet transform, etc. The extracted heartsound signal characteristic includes, but is not limited to, atime-frequency diagram, a Mel spectrum coefficient, a Mel frequencycepstrum coefficient, a zero-crossing rate, a root mean square, aspectral entropy, as well as time-domain waveform characteristics suchas P wave, R wave, T wave, S wave, and the original sound signal.

In Step S530, the heart sound characteristic is categorized using thepre-trained identity recognition model.

The pre-trained identity recognition model may be a binary categorizer.Specifically, the extracted heart sound signal characteristic iscategorized using a categorizer trained according to the registered userdata, and an output tag is obtained. Next, it is determined whether theoutput tag is consistent with the registered user. In this way, theidentity authentication is completed.

In the present disclosure, the identity recognition model can use a deeplearning model such as a support vector machine (SVM) or a convolutionalneural network (CNN) and a recurrent neural network (RNN), or use othermachine learning methods such as random forest, and K-nearest neighboralgorithm (KNN), etc. The present disclosure does not limit types andspecific structures of the identity recognition model.

In the embodiments of the present disclosure, the categorizer is trainedby machine learning for distinguishing the legitimate user fromunauthorized users, which can accurately and quickly perform identityauthentication.

In Step S540, it is determined whether an output category matches with atag of an entered user.

An identity authentication result is determined according to the outputcategory of the identity recognition model. That is, it is returned thelegitimate user (Step S550), or it is returned the illegitimate user(Step S560).

In Step S570, the heart sound characteristic is categorized by using thepre-trained emotion recognition model.

The pre-trained emotion recognition model may be a categorizer thatimplements multiple categorizations. Specifically, the extracted heartsound signal characteristic is categorized using a categorizer trainedaccording to the registered user data or a general-purpose emotioncategorizer built in the system, and output tags are obtained, whereinthe output tags represent different emotional states.

In the present disclosure, the emotion recognition model may use a deeplearning model such as a support vector machine (SVM) or a convolutionalneural network (CNN) and a recurrent neural network (RNN), or use othermachine learning methods such as random forest, and K-nearest neighboralgorithm (KNN), etc. The emotional states include, for example,happiness, sadness, anger, fear, and normal, etc. The present disclosuredoes not limit types and specific structures of the emotion recognitionmodel and categories of the emotional states.

In the embodiments of the present disclosure, the user's emotional stateis obtained by training the categorizer through machine learning, whichcan accurately perform emotion monitoring. It is to be noted that theuser's physiological health can also be monitored based on the heartsound signal characteristic.

In Step S580, the outputted emotion tags are recorded, and an emotionmonitoring report is generated.

The current emotional state of the user is determined and archivedaccording to the output tags of the emotion recognition model togenerate the emotion monitoring report.

It is to be understood that traditional functions (i.e., voiceconversation and music playback, etc.,) of the smart headset provided bythe present disclosure can be implemented by using related technicalsolutions disclosed, which is not unnecessarily elaborated any moreherein.

To further verify effects of the present disclosure, the inventorsimplement the design of a prototype machine. Experiments have provedthat for 50 registered users, this identity authentication method canreach categorization accuracy as high as 95%. The emotion recognitionmodel trained according to the registered user data can reachrecognition accuracy as high as 90% for four types of emotions, and thegeneral-purpose emotion recognition model can reach recognition accuracyas high as 70% for the four types of emotions, which can fully meetrequirements for daily applications.

In summary, the smart headset provided by the present disclosure has thefunctions and shapes of ordinary headsets. However, the smart headsetprovided by the present disclosure can achieve the identityauthentication and emotion monitoring based on a heart sound signalwithout additional electrodes, and thus will not cause any interferenceto the user during the operation of the device. The smart headset islower in hardware costs, convenient to carry and use, and strong inrobustness. As long as the user wears the headset as usual, the identityauthentication and emotion monitoring can be quickly and autonomouslyperformed while the user's privacy is protected, and the headset deviceis unlocked, and even a smart device (such as a mobile phone, acomputer, and a wearable device, etc.) paired with the headset isfurther unlocked. Therefore, the smart headset is suitable for dailyuse.

The present disclosure may be a system, a method, and/or a computerprogram product. The computer program product may include a computerreadable storage medium having computer readable program instructionsthereon for causing a processor to carry out aspects of the presentdisclosure.

The computer readable storage medium may be a tangible device that canretain and store instructions for use by an instruction executiondevice. The computer readable storage medium may be, for example, but isnot limited to, an electronic storage device, a magnetic storage device,an optical storage device, an electromagnetic storage device, asemiconductor storage device, or any suitable combination of theforegoing. More specific examples (a non-exhaustive list) of thecomputer readable storage medium include the following: a portablecomputer diskette, a hard disk, a random access memory (RAM), aread-only memory (ROM), an erasable programmable read-only memory (EPROMor Flash memory), a static random access memory (SRAM), a portablecompact disc read-only memory (CD-ROM), a digital versatile disk (DVD),a memory stick, a floppy disk, a mechanically encoded device such aspunch-cards or raised structures in a groove having instructionsrecorded thereon, and any suitable combination of the foregoing. Thecomputer readable storage medium, as used herein, is not to be construedas being transitory signals per se, such as radio waves or other freelypropagating electromagnetic waves, electromagnetic waves propagatingthrough a waveguide or other transmission media (e.g., light pulsespassing through a fiber-optic cable), or electrical signals transmittedthrough a wire.

Computer readable program instructions described herein can bedownloaded to respective computing/processing devices from a computerreadable storage medium or to an external computer or external storagedevice via a network, for example, the Internet, a local area network, awide area network and/or a wireless network. The network may includecopper transmission cables, optical fiber transmission, wirelesstransmission, routers, firewalls, switches, gateway computers and/oredge servers. A network adapter card or network interface in eachcomputing/processing device receives computer readable programinstructions from the network and forwards the computer readable programinstructions for storage in a computer readable storage medium withinthe respective computing/processing device.

The computer program instructions for carrying out operations of thepresent disclosure may be assembler instructions,instruction-set-architecture (ISA) instructions, machine instructions,machine dependent instructions, microcode, firmware instructions,state-setting data, or either source code or object code written in anycombination of one or more programming languages, including anobject-oriented programming language such as Smalltalk, C++ or the like,and conventional procedural programming languages, such as the “C”programming language or similar programming languages. The computerreadable program instructions may execute entirely on the user'scomputer, partly on the user's computer, as a stand-alone softwarepackage, partly on the user's computer and partly on a remote computeror entirely on the remote computer or server. In a scenario involvedwith the remote computer, the remote computer may be coupled to theuser's computer through any type of network, including a local areanetwork (LAN) or a wide area network (WAN), or may be coupled to anexternal computer (for example, through the Internet using an InternetService Provider). In some embodiments, electronic circuitry including,for example, programmable logic circuitry, field-programmable gatearrays (FPGA), or programmable logic arrays (PLA) may execute thecomputer readable program instructions by utilizing state information ofthe computer readable program instructions to personalize the electroniccircuitry, in order to perform aspects of the present disclosure.

Aspects of the present disclosure are described with reference toflowcharts and/or block diagrams according to the method, apparatus(system) and a computer program product of the embodiments of thepresent disclosure. It is to be understood that each block of theflowcharts and/or block diagrams, and combinations of blocks in theflowcharts and/or block diagrams, can be implemented by the computerreadable program instructions.

These computer readable program instructions may be provided to aprocessor of a general purpose computer, special purpose computer, orother programmable data processing apparatus to produce a machine, suchthat these instructions, which execute via the processor of the computeror other programmable data processing apparatus, create means forimplementing the functions/acts specified in one or more blocks in theflowcharts and/or block diagrams. These computer readable programinstructions may also be stored in a computer readable storage mediumthat can direct a computer, a programmable data processing apparatus,and/or other devices to function in a particular manner, such that thecomputer readable medium having instructions stored therein includes anarticle of manufacture including instructions which implement aspects ofthe function/act specified in one or more blocks in the flowchartsand/or block diagrams.

The computer readable program instructions may also be loaded onto acomputer, other programmable data processing apparatus, or other deviceto cause a series of operational steps to be performed on the computer,other programmable apparatus or other device to produce a computerimplemented process, such that the instructions which execute on thecomputer, other programmable apparatus, or other device implement thefunctions/acts specified in one or more blocks in the flowcharts and/orblock diagrams.

The flowcharts and block diagrams in the accompanying drawingsillustrate architectures, functions and operations of possibleimplementations of systems, methods, and computer program productsaccording to a plurality of embodiments of the present disclosure. Inthis regard, each block in the flowcharts or block diagrams mayrepresent a module, a program segment, or a portion of instructions,which includes one or more executable instructions for implementing thespecified logical function(s). In some alternative implementations, thefunctions denoted in the blocks may occur in a sequence different fromthe sequences shown in the accompanying drawings. For example, twoblocks shown in succession may, in fact, be executed substantiallyconcurrently, or the blocks may sometimes be executed in a reversesequence, depending upon the functions involved. It is also to be notedthat each block in the block diagrams and/or flowcharts and/or acombination of the blocks in the block diagrams and/or flowcharts may beimplemented by a special-purpose hardware-based system executingspecific functions or acts, or by a combination of a special-purposehardware and computer instructions. It is well known to those skilled inthe art that implementations by means of hardware, implementations bymeans of software and implementations by means of software incombination with hardware are equivalent.

The descriptions of the various embodiments of the present disclosurehave been presented above for purposes of illustration, but are notintended to be exhaustive or limited to the embodiments disclosed.Therefore, it is apparent to an ordinary skilled person in the art thatmodifications and variations could be made without departing from thescope and spirit of the embodiments. The terminology used herein ischosen to best explain the principles of the embodiments, the practicalapplication or technical improvement over technologies found in themarketplace, or to enable others of ordinary skill in the art tounderstand the embodiments disclosed herein. The scope of the presentdisclosure is limited by the appended claims.

What is claimed is:
 1. A method for a user recognition and an emotionmonitoring based on a smart headset, wherein the smart headsetcomprising an earplug part and a main body, the earplug part is providedwith a first microphone and a wearing detection sensor, and a housing ofthe main body is internally provided with a signal amplificationcircuit, a communication module, and a microcontroller, wherein themethod comprises detecting whether a user wears the smart headsetproperly by the wearing detection sensor, obtaining a sound signal in anear canal by the first microphone, amplifying the sound signal by thesignal amplification circuit to obtain an amplified sound signal,outputting the amplified sound signal to the microcontroller,transmitting the amplified sound signal by the microcontroller via thecommunication module to a smart terminal paired with the smart headsetto extract a heart sound signal characteristic, validating a legality ofan identity of the user and inferring a current emotional state of theuser according to the heart sound signal characteristic.
 2. The methodaccording to claim 1, wherein the housing of the main body is providedwith a key for activating the first microphone to collect a sound in theear canal.
 3. The method according to claim 1, wherein the smart headsetis an in-ear headset, an ear-hanging headset or a neck-hanging headset.4. The method according to claim 3, wherein the wearing detection sensoris a photoelectric sensor and the wearing detection sensor is arrangedobliquely below an in-ear earplug part.
 5. The method according to claim1, wherein the housing of the main body is provided with one or more ofa tuning key, a power key, a play key, and a pause key.
 6. The methodaccording to claim 1, wherein the housing of the main body is furtherinternally provided with a second microphone configured to implementcommunications.
 7. The method according to claim 1, wherein the earplugpart is further provided with a speaker, and the speaker and the firstmicrophone are of an integrated design.
 8. The method according to claim1, wherein the step of validating the legality of identity of the userand inferring the current emotional state of the user according to theheart sound signal characteristic comprises steps of: collecting anoriginal sound signal in the ear canal by using the first microphonearranged in the earplug part; amplifying the original sound signal toobtain the amplified sound signal in the ear canal; processing theamplified sound signal in the ear canal and extracting the heart soundsignal characteristic, inputting the heart sound signal characteristicinto a pre-trained identity recognition model for an identityauthentication, and inputting the heart sound signal characteristic intoa pre-trained emotion recognition model for an emotional categorization;and determining whether to unlock the smart headset and whether tounlock the smart terminal paired with the smart headset according to anidentity authentication result, and determining the current emotionalstate of the user according to an emotional categorization result togenerate an emotion monitoring report.
 9. The method according to claim8, wherein the step of processing the amplified sound signal in the earcanal and extracting the heart sound signal characteristic comprisessteps of: framing the amplified sound signal in the ear canal by using aHamming window function; denoising the amplified sound signal in eachwindow through filtering to obtain a heart sound signal; and extractinga time-domain characteristic and a frequency-domain characteristic fromthe heart sound signal through a time-frequency transformation, andcombining the time-domain characteristic and the frequency-domaincharacteristic into the heart sound signal characteristic.
 10. Themethod according to claim 9, wherein the step of denoising the amplifiedsound signal in the each window through filtering comprises: denoisingusing a band-pass filter based on a heart rate of the user and asampling frequency of the first microphone.
 11. The method according toclaim 8, wherein a training process of the pre-trained identityrecognition model comprises: collecting a first heart sound signal of aheadset user and a second heart sound signal of a non-user andextracting heart sound characteristics from the first heart sound signaland the second heart sound signal to construct a training sample set;and training an identity recognition model by taking the heart soundcharacteristics of the training sample set as an input and a type of acorresponding user or non-user as an output to obtain a binarycategorizer for distinguishing a legitimate user and an unauthorizeduser.
 12. The method according to claim 8, wherein a training process ofthe pre-trained emotion recognition model comprises: collecting heartsound signals of a headset user in different emotional states andextracting heart sound characteristics from the heart sound signals toconstruct a training sample set; and training an emotion recognitionmodel by taking the heart sound characteristics of the training sampleset as an input and types of corresponding emotional states as an outputto obtain a categorizer for inferring the current emotional state of theuser.
 13. A computer-readable storage medium, storing a computerprogram, wherein when the computer program is executed by a processor,the computer program implements steps of the method according toclaim
 1. 14. An electronic device, comprising a memory and a processor,a computer program capable of running in the processor is stored in thememory, wherein when the processor executes the computer program, stepsof the method according to claim 1 are implemented.
 15. Thecomputer-readable storage medium according to claim 13, wherein thehousing of the main body is provided with a key for activating the firstmicrophone to collect a sound in the ear canal.
 16. Thecomputer-readable storage medium according to claim 13, wherein thesmart headset is an in-ear headset, an ear-hanging headset or aneck-hanging headset.
 17. The computer-readable storage medium accordingto claim 16, wherein the wearing detection sensor is a photoelectricsensor and the wearing detection sensor is arranged obliquely below anin-ear earplug part.
 18. The computer-readable storage medium accordingto claim 13, wherein the housing of the main body is provided with oneor more of a tuning key, a power key, a play key, and a pause key. 19.The computer-readable storage medium according to claim 13, wherein thehousing of the main body is further internally provided with a secondmicrophone configured to implement communications.
 20. Thecomputer-readable storage medium according to claim 13, wherein theearplug part is further provided with a speaker, and the speaker and thefirst microphone are of an integrated design.